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ABSTRACT 

Decision  verification  in  quality  assurance  is  simply  the  Government's 
determination  of  whether  or  not  a  contractor's  inspection  decisions  are 
correct.  This  paper  discusses  procedures  for  accomplishing  decision 
verification  both  as  prescribed  by  directives  and  as  actually  done.  It 
examines  reasons  why  decision  verification  does  not  appear  to  be  a  viable 
element  in  current  quality  assurance  programs  even  though  this  procedure 
is  still  prescribed  in  Navy  directives  for  use  by  contract  administrators, 
The  use  of  operating  characteristic  curves  as  a  managerial  aid  to  deter- 
mine whether  or  not  to  use  decision  verification  is  also  discussed. 
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I.  INTRODUCTION 

Because  of  the  shift  in  emphasis  away  from  "hands-on"  inspection 
by  the  Government  in  quality  assurance,  effective  methods  of  evaluating 
the  contractor's  quality  program  have  been  sought  by  government  agen- 
cies responsible  for  contract  administration.  Decision  verification  is 
one  procedure  which  was  developed  to  assist  in  assessing  the  effective- 
ness of  a  contractor  in  controlling  product  quality. 

Decision  verification  involves  the  Government's  determination  as  to 
whether  contractor  inspection  decisions  are  correct  or  incorrect. 
Statistical  methods  are  then  used  to  determine  if  the  contractor's  in- 
spection process  is  effective  under  criteria  established  by  the  Govern- 
ment. 

Decision  verification  has  been  applied  with  varying  degrees  of  in- 
terest, enthusiasm,  and  success.  Some  of  the  Government  agencies  which 
adopted  these  procedures  in  the  1960 's  have  since  ceased  to  use  them 
because  of  real  or  perceived  difficulties.  In  this  paper  we  shall  be 
concerned  with  what  the  problems  are,  and  how  to  overcome  them  so  that 
decision  verification  can  be  a  useful  means  of  judging  the  inspection 
decision  process  of  the  contractor.  The  basic  approach  is  to: 

1.  Discuss  how  decision  verification  should  be  accomplished  if 
done  in  accordance  with  guidance  contained  in  Navy  directives. 

2.  Discuss  how  decision  verification  was  observed  to  be  done  at 
selected  activities,  if  it  was  done  at  all. 

3.  Show  how  the  specified  decision  verification  procedures  were 
developed,  and  suggest  what  could  be  done  to  improve  the  accomplishment 
of  decision  verification. 


As  an  initial  step  in  research,  applicable  directives,  reports, 
and  other  documents  dealing  with  decision  verification  were  examined. 
This  provided  a  basic  knowledge  of  how  decision  verification  should 
be  done,  at  least  in  theory.  Next,  visits  to  several  production 
facilities  and  interviews  with  quality  assurance  personnel  at  these 
facilities  and  in  other  Government  offices  provided  information  on  how 
decision  verification  really  was  done  and  the  problems  encountered  in 
doing  it. 

The  plan  of  this  paper  is  to  compare  the  specified  with  the  actual. 
Early  discussion  centers  on  the  quality  assurance  policy  of  the  Depart- 
ment of  Defense,  which  caused  procedures  like  decision  verification  to 
be  developed.  Then  the  relationship  of  decision  verification  to  other 
quality  assurance  procedures  in  the  overall  Navy  quality  assurance  pro- 
gram is  discussed.  How  various  decision  verification  procedures  differ 
within  the  Navy  is  also  highlighted.  This  is  followed  by  a  discussion 
of  the  "real"  world  procedures  or  how  decision  verification  is  actually 
done  in  the  field.  The  final  section  of  this  paper  contains  conclu- 
sions and  recommendations. 

Now,  to  set  the  scene,  the  next  chapter  discusses  Department  of 
Defense  quality  assurance  policy. 


II.  DECISION  VERIFICATION  IN  POD 
QUALITY  ASSURANCE  PROGRAMS 

Decision  verification  procedures  are  used  to  determine  whether  or 
not  a  contractor's  inspection  decisions  are  correct  or  incorrect,  and 
with  this  information  in  hand,  to  assess  the  effectiveness  of  the  con- 
tractor's inspection  process.  This  chapter  deals  with  the  application 
of  decision  verification  procedures  by  the  Navy  and  other  organizations 
within  the  Department  of  Defense.  The  purpose  of  this  chapter  is  to 
tell  why  these  procedures  were  developed,  and  where  they  fit  into  the 
quality  assurance  programs  of  various  organizations  within  the  Depart- 
ment of  Defense.  This  chapter  gives  some  reasons  for  the  emphasis  on 
quality  assurance  by  Government  procurement  organizations  and  discusses 
DOD  quality  assurance  policy.  It  outlines  the  basic  elements  of  quality 
assurance  programs  of  two  U.  S.  Navy  organizations  and  highlights  some 
of  the  differences  which  exist  in  these  programs. 

A.   IMPORTANCE  OF  QUALITY  ASSURANCE 

Quality  assurance  is  an  important  part  of  systems  procurement  by 
the  Department  of  Defense.  It  is  easy  to  cite  at  least  three  reasons 
for  this. 

First,  many  of  the  current  systems  purchased  by  DOD  are  highly  com- 
plex. This  is  true  not  only  for  major  systems  such  as  aircraft  and 
ships,  but  also  for  smaller  weapons  systems.  For  example,  the  MK  56 
sea  mine  requires  140  individual  components  for  assembly  of  the  completed 
weapon.  Effective  quality  assurance  improves  the  probability  that  these 
140  components  will  be  assembled  into  a  unit  that  will  work  as  designed. 


The  second  reason  is  closely  allied  to  the  first.  Frequently, 
many  of  the  individual  components  needed  for  the  complete  system  are 
produced  by  different  manufacturers  using  a  variety  of  processes. 
Many  prime  and  sub-contractors  produce  items  which  must  function 
together  in  the  end  product.  For  example,  over  125  different  contrac- 
tors are  manufacturing  hardware  for  the  NASA  Space  Shuttle.  These  com- 
ponents and  subassemblies  range  from  simple  transformers  to  such 
complex  equipment  as  the  Space  Shuttle's  main  engines  [Ref.  1] .  Effec- 
tive quality  assurance  is  needed  in  order  to  minimize  the  problems 
associated  with  assembling  all  of  these  different  components  into  one 
major  vehicle. 

A  third  reason  for  the  emphasis  on  quality  assurance  is  the  avoid- 
ance of  financial  waste  associated  with  systems  which  fail  to  perform 
as  designed.  A  communications  or  intelligence  satellite  program, 
which  can  cost  well  over  $100  million,  may  waste  substantial  resources 
in  support  equipment  and  facilities  if  the  space  vehicle  does  not 
survive  in  orbit  as  long  as  intended. 

B.  DOD  QUALITY  ASSURANCE  PHILOSOPHY  AND  POLICY 

The  current  DOD  philosophy  on  quality  assurance  was  initially  de- 
veloped in  1954  and  formally  expressed  in  DODINST  4155.6.  This  instruc- 
tion made  contractors  responsible  for  assuring  that  material  submitted 
for  acceptance  to  the  Government  did  meet  contract  specifications  and 
requirements.  The  contractor  had  always  been  responsible  for  producing 
material  to  the  requirements  of  the  contract,  but  prior  to  1954  it  had 
been  up  to  the  Government  to  determine  whether  or  not  the  material  was 
acceptable.  Since  acceptability  was  determined  by  inspection  of  the 
material  itself,  a  considerable  amount  of  "hands-on"  inspection  by  the 
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Government  was  necessary.  Beginning  in  1954  there  was  a  redirection 
of  quality  assurance  emphasis  away  from  product  inspection  and  toward 
the  assessment  of  the  contractor's  quality  assurance  program.  The 
Department  of  Defense  no  longer  performed  the  extensive  product  in- 
spections which  had  formerly  characterized  its  approach  to  quality 
assurance.  Instead,  the  contractor  was  now  responsible  for  performing 
many  of  the  product  inspections  previously  done  by  the  Government 
quality  assurance  representative  [Ref .  2] . 

The  basic  DOD  procurement  quality  assurance  policy  can  be  found  in 
Armed  Services  Procurement  Regulations  [Ref.  3] .  This  document  sets 
forth  the  quality  assurance  responsibilities  of  the  Government  and  of 
the  contractor,  and  issues  policy  guidance  for  quality  assurance  rather 
than  detailed  plans  for  establishing  and  operating  an  effective  quality 
program. 

These  regulations  make  it  clear  that  the  Government  must  determine 
the  quality  requirements  which  are  to  be  included  in  the  contract. 
This  is  usually  done  by  contractual  provisions  dealing  with  performance 
specifications,  product  specifications,  standard  inspection  clauses, 
and  other  quality  assurance  related  specifications  such  as  MIL-Q-9858A, 
Quality  Assurance  Program  Requirements. 

The  contractor  is  assigned  responsibility  for  controlling  the  , 
quality  of  his  product  and  for  assuring  that  only  contractually  conform- 
ing items  are  delivered  to  the  Government.  He  must  also  maintain,  and 
furnish  to  the  Government  if  requested,  objective  evidence  that  he  is 
fulfilling  his  quality  responsibilities. 

Armed  Services  Procurement  Regulations  do  not  prescribe  specific 
quality-assurance  measures.  In  accordance  with  these  regulations: 
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It  is  the  contract  administration  office  responsibility  to 
develop  and  apply  effective  and  efficient  procedures  for 
Government  procurement  quality  assurance. 

The  regulations  list  a  number  of  possible  quality  assurance  procedures 

from  which  the  responsible  contract  administration  office  may  choose 

for  administering  the  quality  requirements  of  a  given  contract.  These 

options  include,  but  are  not  limited  to  the  following: 

1.  Physical  inspection  of  the  contract  item. 

2.  Evaluation  and  approval  or  disapproval  of  the  contractor's 
system  for  controlling  quality. 

3.  Use  of  information  from  consumers  to  uncover  deficiencies. 

Now  that  Defense  Department  quality  assurance  policy  has  been  out- 
lined, we  will  discuss  some  of  the  organizations  that  carry  out  that 
policy,  with  particular  emphasis  on  organizations  that  administer  Navy 
contracts . 

Navy  contracts  are  administered  by  one  of  several  sources.  Some 
of  these  contracts  are  administered  by  the  office  of  a  Naval  Plant 
Representative  which  is  located  at  the  contractor's  plant.  Others  are 
administered  by  the  office  of  the  Supervisor  of  Shipbuilding,  Conver- 
sion, and  Repair  located  at  each  Naval  District  headquarters.  Navy  con- 
tracts for  material  which  is  being  produced  in  plants  under  the  primary 
cognizance  of  another  Service  such  as  the  Air  Force  are  normally 
administered  by  that  Service.  The  Defense  Contract  Administration 
Service  often  administers  contracts  for  military  material  production 
in  plants  where  no  Service  personnel  are  assigned. 


~1J.  S.  Department  of  Defense,  Armed  Services  Procurement  Regulations, 
(Washington:  1  July  1976),  p.  14:8. 
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It  is  interesting  to  note  that  Defense  Department  agencies  includ- 
ing the  Navy  are  sometimes  tasked  to  administer  contracts  for  other 
Government  organizations  outside  of  the  Department  of  Defense.  For  ex- 
ample, the  Defense  Contract  Administration  Service  has  been  delegated 
to  act  for  NASA  on  some  Space  Shuttle  contracts.  NASA  does  not  prescribe 
all  of  the  detailed  quality  assurance  actions  which  the  delegated  agency 
should  take.  The  Defense  Contract  Administration  Service  must  use  its 
existing  procedures  as  well  as  any  guidance  or  special  instructions  which 
NASA  provides  [Ref .  4] . 

C.  SOME  DIFFERENCES  IN  QUALITY  ASSURANCE 
PROGRAMS  OF  VARIOUS  AGENCIES 

It  has  been  pointed  out  that  Navy  contracts  are  sometimes  adminis- 
tered by  other  agencies,  and  that  in  turn  the  Navy  may  be  called  upon 
to  administer  other  than  Navy  contracts.  Since  the  contract  administra- 
tion activity  generally  uses  its  own  quality  assurance  procedures  on 
all  contracts  that  it  administers,  it  is  important  to  note  some  of  the 
differences  that  exist  between  the  quality  assurance  programs  of  the 
various  DOD  agencies. 

Requirements  for  DOD  quality  assurance  programs  are  contained  in 
Defense  Supply  Agency  Manual  8200.1  of  August  1976  which  is  entitled, 
Procurement  Quality  Assurance  [Ref.  5] .  Although  this  is  a  Joint  Ser- 
vice directive  and  was  co-signed  by  representatives  of  each  military 
Service,  the  Navy  has  not  yet  chosen  to  abide  by  this  manual  without 
issuing  its  own  supplemental  instructions. 

In  connection  with  this  discussion  of  decision  verification,  it  is 
interesting  to  note  that  one  of  the  predecessors  of  the  present  manual 
was  issued  in  1969.  This  directive,  commonly  referred  to  as  Handbook 
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57,  contained  a  decision  verification  procedure  called  "Contractor 
Decision  Verification"  [Ref .  6] .  The  requirement  to  accomplish  Contrac- 
tor Decision  Verification  was  cancelled  in  1973  when  Handbook  57  was 
superseded  by  an  early  version  of  Defense  Supply  Agency  Manual  8200.1. 
Consequently,  the  Defense  Contract  Administration  Service,  which  organi- 
zationally is  part  of  the  Defense  Supply  Agency,  no  longer  uses 
Contractor  Decision  Verification  procedures  in  administering  contracts. 
Thus  we  see  that  a  very  important  part  of  the  DOD  contract  administra- 
tion organization  omits  decision  verification  as  an  element  in  quality 
assurance. 

Unlike  the  Defense  Contract  Administration  Service,  the  Navy  has 
not  eliminated  decision  verification  from  its  repertory  of  quality- 
assurance  procedures.  At  least  two  organizations  within  the  Navy 
material  headquarters  have  issued  quality  assurance  instructions  con- 
taining decision  verification  procedures  to  be  used  by  their  field 
contract-administration  offices . 

The  Naval  Air  Systems  Command  is  one  of  the  two  organizations. 
It  exercises  control  over  contract  administration  of  production  in 
some  contractors'  plants  through  the  office  of  a  Naval  Plant  Represen- 
tative or  NAVPRO  who  is  physically  located  at  the  production  facility. 
He  is  responsible  for  developing  and  applying  effective  and  efficient 
quality  assurance  plans  and  procedures  which  are  consistent  with 
requirements  of  the  particular  contracts  which  he  administers. 

The  basic  elements  of  the  quality  assurance  program  for  use  by 
contract-administration  field  activities  which  are  under  the  cognizance 
of  the  Naval  Air  Systems  Command  are  as  follows: 
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1.  Planning.  The  NAVPRO  initiates  planning  as  soon  as  the  con- 
tract document  is  received.  Planning  includes  the  selection  of  the 
applicable  elements  which  are  compatible  with  local  production  condi- 
tions. 

2.  Procedures  Review.  A  review  is  conducted  to  confirm  that  the 
contractor's  quality  procedures  are  complete  and  comprehensive  in 
scope. 

3.  Procedures  Evaluation.  A  continuing  assessment  is  made  of  the 
contractor's  procedures  to  confirm  that  his  procedures  are  adequate 

to  assure  the  quality  of  the  products  or  services  delivered  to  the 
Government,  and  that  the  contractor  is  actually  following  these  pro- 
cedures . 

4.  Product  Verification.  The  actual  product  item  is  examined, 
tested,  and  inspected  to  determine  whether  or  not  the  item  conforms 
to  the  contract's  requirements. 

5.  Contractor  Decision  Verification.  Decision  verification  pro- 
cedures are  a  primary  topic  of  this  thesis.  Contractor  Decision 
Verification  is  defined  as  "A  random  and  continuing  evaluation  of  the 

contractor's  decisions,  performed  by  comparing  the  findings  of  Govern- 

2 
ment  inspection  with  contractor's  inspection  records."   It  will  be 

applied  whenever  "in-process  product  control  is  performed  by. the  con- 
tractor and  the  volume  of  production  and  duration  of  contracts  is 
stable."3 


2Naval  Air  Systems  Command,  NAVAIR  FIELD  CONTRACT  MANUAL,  NAVAIRINST 
4330.16  (Washington:   Nov.  1971),  p.  4-108. 

3 
Ibid. ,  p.  4-5. 
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6.  Quality  Data  Evaluation.  A  periodic  evaluation  of  all  accumu- 
lated quality  data  is  made  for  the  specific  contract  being  administered. 
This  data  includes  such  items  as  contractor  inspection  records,  pro- 
cedures review  information,  and  user  feedback  on  the  quality  of  the 
product.  It  is  used  for  a  number  of  purposes  including  adjustment  of 
the  intensity  of  Government  inspection. 

7.  Corrective  Action.  Corrective  action  is  the  action  that  the 
Government  requires  the  contractor  to  take  in  order  to  correct  defi- 
ciencies in  the  product  delivered  to  the  Government  [Ref.  7], 

Thus  we  see  that  Contractor  Decision  Verification  is  one  of 
several  quality  assurance  techniques  which  the  NAVPRO  can  use  in  his 
quality  assurance  program. 

Another  Navy  organization  which  still  retains  decision  verifi- 
cation procedures  as  part  of  its  quality  assurance  program  is  the  Naval 
Sea  Systems  Command.  This  command  has  issued  two  manuals.  One  is 
applicable  to  ship  acquisition  contracts,  and  the  other  manual  applies 
to  ship  repair  contracts  [Ref s .  8,9]. 

The  quality  program  for  ship  acquisition  is  similar  to  that  for 
ship  repair  contracts,  but  there  are  significant  differences  between 
the  two.  For  example,  the  basic  elements  which  constitute  the  procure- 
ment quality  assurance  program  for  ship  repair  contracts  are  also 
applicable  to  the  quality  assurance  program  for  ship  acquisition  con- 
tracts, but  an  additional  element  is  included  in  the  latter.  The 
following  is  a  list  of  the  basic  elements  of  both  the  ship  acquisition 
quality  program  and  the  ship  repair  contract  quality  program.  Elements 
common  to  both  programs  are  indicated  by  an  asterisk. 
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1.  Plaining." 

2 .  Procedures  Review. * 

3.  Product  Verification" 

4.  Verification  of  Contractor's  Records.  This  procedure  is  in- 
cluded in  the  quality  assurance  program  for  ship  acquisition  but  not 
for  ship  repair.  It  is  a  decision  verification  procedure  similar  to 
Contractor  Decision  Verification  procedures  previously  mentioned  in 
the  discussion  about  quality  programs  of  the  Naval  Air  Systems  Command. 

5.  Procedures  Evaluation" 

6.  Corrective  Action" 

7.  Quality  Data  Evaluation" 

From  this  listing  we  see  that  verification  of  contractor's  records 
is  an  element  of  quality  assurance  programs  for  ship  acquisition,  but 
not  for  ship  repair  contracts.  It  should  be  pointed  out  here  that  a 
much  stronger  emphasis  on  the  independence  between  Government  quality- 
assurance  functions  and  contractor  quality  functions  is  found  in  the 
quality  assurance  instructions  for  ship  repair  contracts,  than  one  can 
find  in  the  quality  ■assurance  directives  for  ship  acquisition.  The 
following  quotation  which  refers  to  ship  repair  contracts  illustrates 
this. 

SUPSHIP  quality  assurance  personnel  shall  not  serve  as  an 
adjunct  to,  or  a  replacement  for  the  contractor's  own  inspection 
system.  For  example,  the  SUPSHIP  office  should  not  serve  as  a 
convenient  means  of  advising  the  contractor  of  obvious  deficiencies 
in  work  performance;  nor  should  it  be  used  by  the  contractor  as  a 
progressive  inspection  device  to  determine  whether  or  not  the  end 
product  will  be  acceptable.14 


4 

Naval  Sea  Systems  Command,  Ship  Repair  Contracting  Manual, 

NAVSEA  0900-LP-079-5010   (Washington:      1974),  p.   1202. 
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Besides  differences  of  philosophy  within  the  Naval  Air  Systems  Com- 
mand, there  are  some  specific  differences  concerning  decision  verifica- 
tion procedures  between  the  Naval  Air  Systems  Command  and  the  Naval  Sea 
Systems  Command.  For  example,  verification  should  be  accomplished  by 
Naval  Air  Systems  Command  contract  administration  activities,  but  is 
optional  for  Naval  Sea  Systems  Command  contract  administration  activities, 

This  chapter  has  shown  that  decision  verification  procedures  are 
still  an  element  of  some  agencies'  quality  assurance  programs.  This 
along  with  the  fact  that  various  agencies  have  somewhat  different 
quality  assurance  -programs  are  the  two  most  important  points  to  be 
gleaned  from  this  chapter.  The  next  chapter  will  discuss  the  concept 
and  objectives  of  decision  verification  in  a  quality  assurance  program. 
It  will  also  describe  specific  procedures  used  by  two  Navy  organizations 
to  accomplish  decision  verification. 
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III.  DECISION  VERIFICATION, 
CONCEPTS  AND  PROCEDURES 


This  chapter  discusses  the  concept  of  decision  verification  and 
the  reasons  for  applying  decision  verification  procedures  in  the  ad- 
ministration of  Navy  contracts,  in  order  to  show  why  these  procedures 
are  used  and  how  they  are  applied.  To  do  this,  the  initial  discus- 
sion will  focus  on  instances  where  decision  verification  can  be  used 
advantageously.  The  discussion  will  then  shift  to  a  narrative  descrip- 
tion of  how  decision  verification  is  to  be  accomplished  when  it  is 
done  in  accordance  with  Navy  directives. 

A.  THE  NEED  FOR  DECISION  VERIFICATION 

In  many  procurements,  there  are  circumstances  in  which  the  examina- 
tion of  the  end  product  alone  cannot  be  used  to  ascertain  whether  or 
not  the  item  is  acceptable  and  meets  contractual  requirements.  When 
large  numbers  of  identical  items  such  as  gun  projectiles  are  manufac- 
tured, it  may  be  too  costly  to  inspect  each  item.  When  expensive 
items  are  subject  to  destructive  testing,  it  may  be  uneconomical  even 
to  test  large  samples  of  the  end  item.  An  aircraft  contains  large 
numbers  of  electrical  cable  runs,  some  of  which  are  inaccessible  for 
normal  inspection  once  the  aircraft  has  been  assembled.  This  wiring 
must  be  installed  in  accordance  with  specifications,  but  it  may  not  be 
feasible  to  perform  100%  product  inspection  on  every  run  of  wiring. 

Since  extensive  product  inspection  is  often  impractical,  it 
appeared  highly  desirable  to  the  authors  of  DOD  quality  assurance 
programs  that  the  Government  be  able  to  determine  that  the  contractor's 
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in-process  operations  are  producing  product  which  is  within  acceptable 
quality  limits.  They  assumed  that  if  the  contractor  has  an  effective 
quality  assurance  program,  his  product  would  meet  contractual  standards 
most  of  the  time.  The  contractor's  inspection  process  is  one  important 
part  of  his  quality  assurance  program  [6 ] .  Therefore  it  is  important 
to  know  how  well  this  process  is  working. 

One  element  of  inspection  is  the  actual  decision  of  the  inspector 
concerning  whether  or  not  a  certain  attribute  or  characteristic  meets 
contractual  requirements.  Measuring  the  effectiveness  of  a  contractor's 
program  by  an  examination  of  these  decisions  will  be  the  subject  of  the 
next  section  of  this  chapter. 

B.  THE  CONCEPT  OF  DECISION  VERIFICATION 

Decision  verification  procedures  originated  from  a  concept  developed 
by  the  Air  Force  in  the  late  1950's.  By  1969  these  procedures  had  been 
adopted  by  the  Defense  Supply  Agency  and  included  in  its  directives  on 
quality  assurance. 

Under  the  concept  of  decision  verification,  the  Government  quality 
assurance  representative  "inspects"  the  decision  made  by  the  contractor's 
inspector.  He  then  determines  whether  the  contractor's  inspector  made 
a  good  or  a  bad  decision.  The  contractor's  inspector  can  make  one  of 
two  basic -errors  which  result  in  a  wrong  decision.  The  inspector  can 
fail  to  report  a  real  defect, or  he  can  report  a  false  defect.  The  former 
happens  when  the  inspector  fails  to  see  a  defect,  or  if  he  does  see  it, 
fails  to  note  it  as  a  defect  through  an  error  in  judgment.  A  false 
detection  is  solely  an  error  in  the  inspector's  judgment.  Therefore, 
the  QA  representative  can  check  on  the  decision  of  the  inspector  using 
both  material  which  is  accepted  by  the  contractor's  inspector  and 
material  which  is  rejected  [Ref .  10] . 
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The  effectiveness  of  the  contractor's  inspection  decisions  can  be 
described  as  a  process  average  representing  the  percent  of  decisions 
that  are  incorrect  because  of  either  type  of  error.  Typically,  direc- 
tives specify  an  acceptable  quality  level  (AQL)  of  1%  for  the  con- 
tractor's decision  process,  which  means  that  the  Government  expects 
that  the  contractor's  inspector  will  make  the  correct  decision  at 
least  99%  of  the  time  [6] . 

It  is  easy  to  see  why  the  AQL  had  to  be  greater  than  zero,  for 
seldom  does  an  inspector  always  make  correct  decisions.  Apparently  the 
1%  AQL  was  chosen  after  evaluating  the  quality  performance  of  a  large 
number  of  contractors  of  various  sizes  producing  various  products. 
Useful  information  as  to  why  an  AQL  of  1%  was  selected  can  be  found 
in  a  1969  Defense  Contract  Administration  Service  lesson  plan  for  Con- 
tractor Decision  Verification  procedures  training  [Ref .  11] .  Accord- 
ing to  this  lesson  plan,  experience  indicates  that  95  percent  of  the 
contractors  will  operate  with  0%  to  1%  process  average  as  related  to 
Contractor  Decision  Verification. 

Decision  verification  was  developed  primarily  to  assure  that  the 
contractor's  quality  program  or  inspection  system  satisfactorily  con- 
trols the  quality  of  the  product.  This  statement  has  occasionally 
been  interpreted  incorrectly  to  mean  that  if  the  contractor's  decision 
process  is  within  acceptable  limits  (1%)  no  quality  defects  should  be 
found  in  the  end  product.  In  March  1971,  the  report  of  a  Joint  Service 
Panel  on  Contract  Administration  strongly  questioned  the  value  of 
decision  verification.  The  report  said: 

In  actual  practice,  rarely  does  a  contractor's  percent  defec- 
tiveness exceed  the  process  average  limits.  As  a  consequence,  in 
some  cases  when  product  quality  deficiencies  have  been  encountered 
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during  product  verification,  the  contractor's  decision  making 
process  is  considered  acceptable  because  it  has  not  exceeded  the 
process  average  limits.  For  this  reason  the  value  of  CDV  as 
currently  applied  is  questioned. ^ 

Overlooked  here  is  the  fact  that  if  the  contractor's  inspection 

system  is  inadequate,  his  inspectors  could  still  make  predominantly 

good  decisions  and  the  end  product  could  still  be  unsatisfactory. 

Decision  verification  procedures  used  by  DOD  really  evaluate  the 

effectiveness  of  the  implementation  of  whatever  inspection  system 

the  contractor  has  installed.  These  procedures  do  not  alone  evaluate 

the  adequacy  of  that  inspection  system,  but  can  be  used  with  other 

methods  to  do  so. 

C.  NAVAL  AIR  SYSTEMS  COMMAND 

DECISION  VERIFICATION  PROCEDURES 

Contract  administration  offices  under  the  Naval  Air  Systems  Com- 
mand have  been  directed  to  use  a  procedure  called  Contractor  Decision 
Verification  in  programs  for  evaluating  a  contractor's  quality  per- 
formance [7],  Contractor  Decision  Verification  is  defined  as  a 
"system  performed  by  the  Government  to  determine  how  well  or  the 
accuracy  at  which  the  contractor's  overall  inspection  operation  is 
functioning."   This  is  accomplished  by  evaluating  the  accuracy  of 
randomly  selected  contractor's  inspection  decisions  on  specific  or 
general  product  quality  characteristics.  The  number  and  type  of 
characteristics  to  be  observed  by  the  Government  quality-assurance 
representative  in  evaluating  the  contractor's  program,  depend  on  the 


5U.  S.  Department  of  Defense,  Report  of  the  Joint  AMC/NMC/AFLC/AFSC 
Panel  on  Contract  Administration  (Washington:  24  Mar  1971),  p.  41. 

c 

Naval  Air  Systems  Command,  Navair  Field  Contract  Administration 
Manual  NAVATRINST  4330.16  (Washington:  Nov  71),  p.  4-105. 
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particular  process  and  product  being  inspected.  Random  selection  is 
done  by  use  of  a  random  number  table. 

Before  actually  carrying  out  verification  procedures,  the  Government 
quality  assurance  representative  groups  the  contractor's  inspection 
activities  into  manageable  units,  and  determines  points  in  the  produc- 
tion flow  process  at  which  the  contractor's  inspection  decisions  should 
be  verified.  These  are  called  verification  stations.  Verification 
stations  should  be  chosen  to  provide  an  equal  chance  for  all  important 
contractor  inspection  decisions  to  be  verified  while  simultaneously  re- 
ducing the  possibility  of  undue  concentration  on  some  particular  process, 
product,  or  individual  inspector.  Once  the  total  number  of  verification 
stations  necessary  for  evaluating  the  contractor's  decision  process  has 
been  determined,  they  are  grouped  into  Product  Control  Centers  contain- 
ing from  three  to  nine  stations  each. 

Establishing  verification  stations  and  Product  Control  Centers  and 
determining  which  product  characteristics  will  be  used  are  NAVPRO  func- 
tions. The  actual  procedures  to  be  followed  in  accomplishing  Contractor 
Decision  Verification  are: 

1.  Select  one-half,  but  not  less  than  three,  verification  stations 
within  each  PCC  using  a  table  of  random  numbers.  Any  station  or  sta- 
tions which  have  not  been  verified  within  three  consecutive  verification 
cycles  are  automatically  included  in  the  next  succeeding  cycle. 

2.  After  selecting  verification  stations,  the  QA  representative 
verifies  that  the  procedures  used  by  the  contractor  in  product  process- 
ing in  each  Product  Control  Center  conform  to  contract  specifications. 
Any  defect  noted  counts  as  one  defective  observation. 

3.  The  next  step  is  to  take  50  observations  per  day  from  each  Prod- 
uct Control  Center  until  a  total  of  250  observations  have  been 
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accumulated  there.  If  there  are  seven  or  fewer  defective  decisions  in 
this  sample  of  250  observations,  the  contractor's  decision  process  is 
considered  to  be  under  control  at  that  center.  The  critical  value  of 
seven  is  based  on  three  sigma  limits.  If  the  contractor's  process  is 
under  control,  the  quality  assurance  representative  shifts  to  less  fre- 
quent sampling  as  discussed  below. 

4.  If  the  contractor's  decision  process  is  under  control,  the 
quality-assurance  representative  takes  50  observations  from  each  PCC 
within  each  subsequent  interval  of  11  calendar  days.   (For  each  11-day 
interval  a  new  random  selection  of  verification  stations  is  made  to 
begin  the  new  cycle.)  At  this  sampling  frequency,  the  process  is  con- 
sidered to  be  under  control  as  long  as  the  percent  defective  in  a 
sample  of  size  50  is  5.2%  or  less.  If  the  percent  defective  exceeds 
5.2%  for  50  observations,  an  additional  50  observations  are  taken  at 
the  same  verification  stations  within  two  days.  If  the  percent  defec- 
tive of  the  total  100  decisions  examined  exceeds  4-. 2%,  the  QA  repre- 
sentative reverts  to  taking  50  observations  per  day  for  five  days  until 
the  contractor  has  taken  effective  corrective  action  and  the  process  is 
again  under  control  [7].  It  should  be  noted  here,  that  these  pro- 
cedures contain  no  provisions  for  using  samples  of  less  than  50  obser- 
vations . 

Thus  we  see  that  Contractor  Decision  Verification  procedures  involve 
an  initial  step  of  taking  50  observations  per  day  until  250  have  been 
accumulated.  Then  if  the  process  is  considered  under  control,  based 
on  criteria  set  forth  in  applicable  directives,  the  quality  assurance 
representative  is  allowed  to  use  a  smaller  sample  at  greater  intervals 
of  time  as  long  as  the  process  remains  under  control.  If  at  any  time 
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during  this  reduced  sampling  period,  the  process  appears  out  of  control 
the  quality  assurance  representative  has  prescribed  procedures  to  fol- 
low to  insure  that  adequate  sampling  is  done  to  confirm  whether  or  not 
the  process  is  in  fact  out  of  control. 

D.  NAVAL  SEA  SYSTEMS  COMMAND  DECISION 
VERIFICATION  PROCEDURES 

The  decision  verification  procedures  used  by  Naval  Sea  Systems  Com- 
mand activities  are  called  Verification  of  Contractor  Records  [8] . 
They  are  analogous  to  Contractor  Decision  Verification  procedures  used 
by  the  Naval  Air  Systems  Command. 

The  Naval  Sea  Systems  Command  seems  to  be  more  flexible  in  the 
application  of  decision  verification  procedures  than  other  organiza- 
tions. It  gives  the  individual  contract  administration  office  con- 
siderable discretion  in  deciding  whether  or  not  to  use  decision  veri- 
fication in  administering  the  quality  assurance  program  associated 
with  any  particular  contract.  One  important  procedural  difference 
between  the  accomplishment  of  Verification  of  Contractor  Records  and 
Contractor  Decision  Verification  should  be  emphasized.  For  verifica- 
tion purposes  the  nominal  sample  size  specified  by  Naval  Air  Systems 
Command  directives  is  50  observations.  Naval  Sea  Systems  Command 
directives  allow  the  local  quality  representative  to  select  the  nominal 
sample  size.  The  effect  of  this  coupled  with  the  use  of  a  constant 
percent  defective  as  the  basis  for  rejecting  the  contractor's  process 
will  be  more  fully  explored  in  later  sections  of  this  paper. 

The  selection  of  stations  for  verification  and  their  grouping  into 
control  centers  is  essentially  the  same  as  done  for  Contractor  Decision 
Verification.  The  number  of  verification  stations  in  each  control 
center  is  three  to  nine. 
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The  following  describes  the  procedures  for  accomplishing  Verifica- 
tion of  Contractor  Records: 

1.  Just  as  is  done  in  Contractor  Decision  Verification,  one-half 
but  not  less  than  three,  verification  stations  from  each  control  center 
are  selected  at  random  using  a  table  of  random  numbers.  Any  station 
or  stations  which  have  not  been  verified  within  three  consecutive  veri- 
fication cycles  are  automatically  included  in  the  next  succeeding 
cycle. 

2.  Also  as  is  done  in  Contractor  Decision  Verification,  one  pro- 
cedural observation  will  be  made  for  each  type  of  product  inspected 
and  included  in  the  sample  of  observations  of  the  applicable  control 
center. 

3.  From  this  point  in  the  procedures,  the  differences  between 
Verification  of  Contractor  Records  and  Contractor  Decision  Verification 
become  significant.  Instead  of  a  specified  sample  of  50  observations 
per  day  from  each  control  center,  the  quality  assurance  representative 
following  decision  verification  instructions  of  the  Naval  Sea  Systems 
Command  determines  locally  the  size  of  the  daily  sample  to  be  taken  in 
each  control  center.  When  five  samples  of  this  established  size  have 
been  accumulated,  the  contractor's  process  is  accepted  as  under  control, 
at  that  center  if  the  percent  defectives  in  the  total  number  of  obser- 
vations in  the  five  groups  is  2.8%  or  less. 

4.  If  at  this  point  the  contractor's  process  is  in  control,  the 
quality  assurance  representative  reduces  the  frequency  of  sampling. 

He  takes  one  sample  of  the  locally  established  size  within  each  seven- 
day  interval.  When  the  percent  defective  of  this  sample  exceeds  2.8% 
regardless  of  the  sample  size,  the  quality  assurance  representative 
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reverts  to  sampling  as  specified  in  the  preceding  paragraph,  until  cor- 
rective action  has  been  taken  by  the  contractor  and  the  decision  process 
has  been  brought  under  control  at  the  1%  AQL  [8] . 

Before  proceeding  to  the  next  topic  it  is  important  to  emphasize 
that  the  allowable  percent  defective  under  Verification  of  Contractor 
Records  procedures  is  a  constant  2.8%  regardless  of  the  size  of  the 
sample,  while  for  Contractor  Decision  Verification  the  limiting  percent 
defective  varies  with  the  size  of  the  sample.  This  will  be  discussed 
in  more  detail  in  Chapter  V. 

E.  POTENTIAL  BENEFITS  OF  AN  EFFECTIVE 
SURVEILLANCE  PROCEDURE 

The  importance  of  quality  assurance  has  already  been  discussed  in 
Chapter  II.  The  quality  assurance  functions  of  DOD  agencies  currently 
consume  a  large  portion  of  contract  administration  resources,  much  of 
which  is  spent  on  product  inspection.  Over  9,000  people,  34  percent  of 
contract  administration  personnel  in  DOD,  are  assigned  quality  assurance 
responsibilities.  A  recent  study  of  DOD  contract  administration  noted 
that  the  Government  was  unnecessarily  duplicating  quality-assurance 
actions  of  the  contractor.  Too  much  effort  was  being  expended  on  prod- 
uct inspection.  For  example,  in  one  geographical  region  studied, 
quality  assurance  personnel  spent  46%  of  their  time  doing  product  inspec- 
tion and  only  22%  of  their  time  doing  surveillance  of  the  contractor's 
quality  assurance  program  [Ref .  12] . 

Decision  verification  procedures  originated  from  the  requirement  to 
reduce  the  need  for  doing  product  inspection.  They  were  developed  as 
surveillance  methods  to  aid  in  determining  the  effectivenss  of  the  con- 
tractor's quality  program.  It  has  been  estimated  that  halving  the 
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current  production  inspection  effort,  even  if  the  man-years  of  effort 
in  surveillance  were  doubled,  would  still  result  in  a  15  to  20  percent 
savings  in  personnel  cost  of  approximately  $15  million  annually  [12] . 

From  the  above  discussion,  it  seems  that  any  workable  surveillance 
method  has  merit.  That  is  why  Contractor  Decision  Verification  pro- 
cedures and  other  similar  procedures  were  developed.  This  chapter  has 
shown  how  these  procedures  should  be  accomplished.  The  following 
chapter  will  relate  how  decision  verification  has  been  implemented  at 
selected  activities. 
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IV.  DECISION  VERIFICATION 
AT  SELECTED  ACTIVITIES 

The  decision  verification  procedures  described  by  the  Naval  Air 
Systems  Command  and  Naval  Sea  Systems  Command  were  discussed  in  the 
previous  chapter,  and  we  now  turn  our  attention  to  the  current  appli- 
cation of  these  procedures  where  they  are  employed.  (Some  contract 
administration  offices  do  not  use  the  decision  verification  procedures 
which  are  specified  in  the  directives  of  the  various  Naval  Systems 
Commands.)  The  purpose  of  this  chapter  is  to  show  the  diversity  which 
exists  in  the  application  of  decision  verification  at  several  field 
activities . 

The  chapter  opens  with  a  description  of  some  of  the  facilities 
which  were  visited  to  gather  information  on  decision  verification  pro- 
cedures. Then  we  will  describe  how  these  procedures  are  done  at  these 
facilities,  and  relate  some  views  about  decision  verification  as 
expressed  by  the  personnel  who  must  use  this  quality  assurance  method. 

A.  DESCRIPTION  OF  THE  FACILITIES  VISITED 

Several  contract  administration  offices  were  visited  to  obtain 
first-hand  information  on  how  decision  verification  procedures  were 
applied  as  part  of  on-going  quality  control  programs.  We  shall  refer 
to  the  activities  visited  as  Facilities  A,  B,  and  C. 

Facility  A  is  located  at  a  large  manufacturing  plant  now  engaged 
in  producing  combat  jet  aircraft  for  the  Navy  as  well  as  for  foreign 
countries.  This  is  the  only  military  aircraft  being  produced  at  this 
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plant.  Although  this  particular  aircraft  has  been  produced  by  this  com- 
pany for  many  years,  production  was  at  a  relatively  low  level  at  the 
time  of  the  visit.  Commercial  products  and  aircraft  are  also  produced 
at  the  same  plant.  A  resident  Naval  Plant  Representative  administers 
Navy  contracts  at  Facility  A. 

Facility  B  is  also  at  a  large  plant  engaged  in  producing  Naval  air- 
craft. Both  jet  and  non-jet  aircraft  are  produced  for  the  Navy  and 
for  foreign  countries,  and  a  resident  Naval  Plant  Representative  admin- 
isters contracts  at  this  plant. 

Facility  C  is  the  office  of  a  Naval  District  Supervisor  of  Ship- 
building, Conversion  and  Repair.  The  quality  assurance  personnel 
assigned  have  responsibilities  at  several  large  and  small  shipyards,  in- 
cluding a  major  yard  now  constructing  two  large  auxiliary  ships  for  the 
United  States  Navy.  This  yard  will  soon  begin  building  a  number  of 
small  combatants.  The  various  production  sites  for  which  this  office 
is  responsible  include  some  involved  in  one-of-a-kind  experimental 
model  production. 

Other  activities  visited  or  contacted  by  phone  included  additional 
Naval  District  SUPSHIP  offices,  and  a  plant  where  NASA  contracts  are 
administered  by  the  Defense  Contract  Administration  Service. 

B.  FINDINGS  AT  FACILITY  A 

The  visit  to  Facility  A  included  interviews  with  the  Naval  Plant 
Representative  and  Government  quality  assurance  personnel. 

The  Naval  Plant  Representative  expressed  the  view  that  the  real 
objective  of  the  Navy  quality  assurance  organization  was  not  to  empha- 
size product  inspection,  but  rather  to  insure  the  effectiveness  of  the 
contractor's  quality  program.  He  was  quite  aware  of  Contractor  Decision 
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Verification,  but  he  indicated  that  the  procedures  as  specified  by 
Naval  Air  Systems  Command  directive,  were  difficult  to  follow  at  this 
plant  because  of  the  nature  of  production.  Because  of  this,  he  had 
been  and  was  interested  in  determining  two  things  related  to  Contractor 
Decision  Verification.  Since  he  was  often  unable  to  acquire  the 
required  sample  size,  he  wanted  to  know  how  many  contractor's  inspec- 
tion decisions  must  be  observed  in  order  to  obtain  useful  results.  He 
also  wanted  to  know  whether  or  not  all  of  the  inspection  decisions 
verified  should  be  decisions  on  military  hardware  since  the  directive 
does  not  specify  this. 

Government  quality  assurance  personnel  with  whom  Contractor  Deci- 
sion Verification  procedures  were  discussed  generally  accepted  them 
as  reasonable,  but  thought  them  excessively  complex.  They  indicated 
that  it  was  difficult  to  use  the  procedures  under  certain  production 
conditions  such  as  low  volume.  Another  important  point  which  emerged 
from  interviews  here,  and  again  in  conversations  with  experienced 
supervisors  at  other  facilities,  was  that  successful  use  of  these  pro- 
cedures depended  largely  upon  the  motivation  and  competence  of  Govern- 
ment quality  assurance  personnel.  Concern  was  expressed  that  some 
personnel  might  not  follow  decision  verification  procedures  correctly 
in  order  to  avoid  personal  conflict  with  contractor  personnel.  Such 
a  situation  could  arise  particularly  in  cases  of  close  association 
between  the  QA  representative  and  the  contractor's  inspector.  It 
appeared  that  conducting  product  inspection  was  much  easier  from  a 
human  relations  standpoint  than  passing  judgment  on  the  competence  of 
individual  contractor's  inspectors  by  accepting  or  rejecting  their 
decisions. 
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The  Naval  Plant  Representative  at  Facility  A  has  had  his  quality- 
assurance  people  study  the  application  of  Contractor  Decision  Verifi- 
cation procedures  as  an  element  of  his  quality  assurance  program.  He 
then  directed  his  people  in  attempting  to  implement  Contractor  Decision 
Verification  as  prescribed  in  Naval  Air  Systems  Command  directives. 
They  then  studied  the  results  to  uncover  problems. 

In  doing  this  trial,  the  quality  assurance  representative  randomly 
selected  at  least  three  verification  stations  from  each  Product  Control 
Center.  He  then  went  to  those  stations  to  inspect  military  hardware 
which  might  be  available  there.  He  frequently  found  that  no  military 
hardware  was  ready  for  inspection,  or  at  least  not  enough  of  it  to  make 
the  required  50  observations  per  day  in  the  control  center. 

Auditing  verification  also  presents  problems.  In  actual  practice, 
results  of  any  observations  that  are  made  are  to  be  recorded  on 
DD  Form  1711  in  accordance  with  prescribed  procedures.  The  characteris- 
tics inspected  are  not  identified  on  this  form.  Instead,  the  verifica- 
tion station  is  noted  on  the  form  together  with  the  number  of  observa- 
tions made  at  each  station.  This  makes  it  difficult  to  audit  results 
in  terms  of  actual  hardware  inspected  unless  a  defect  is  noted,  and 
corrective  action  for  that  specific  hardware  deficiency  is  initiated. 

Facility  A  currently  does  not  perform  decision  verification ■ pro- 
cedures in  the  manner  prescribed  by  directives  because  the  requirement 
for  decision  verification  can  be  waived  if  the  volume  of  production  is 
insufficient  to  support  its  accomplishment.  Locally  prepared  quality 
assurance  directives,  Quality  Division  Operating  Procedures,  state  that 
"statistical"  methods  of  accomplishing  decision  verification  have  proven 
unworkable,  but  recognize  the  concept  of  decision  verification  as  a 
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reasonable  and  important  part  of  procurement  quality  assurance.  Conse- 
quently, modified  decision  verification  procedures  have  been  developed 
for  use  at  Facility  A.  Under  the  previously  mentioned  local  directives, 
the  quality  assurance  representative  evaluates  the  correctness  of  the 
contractor's  acceptance  decision  by  verifying  one  or  more  products  in 
each  Product  Control  Center.  The  section  supervisor  determines  when  the 
total  number  of  verifications  is  adequate  in  each  case. 

By  way  of  review  for  a  moment,  recall  that  the  purpose  of  decision 
verification  is  to  determine  the  effectiveness  of  the  contractor's  in- 
spection decisions,  and  that  this  effectiveness  can  be  expressed  as  a 
process  average  in  terms  of  the  percent  of  decisions  that  are  defective. 
The  contractor's  decision  process  average  is  compared  with  an  established 
percent  defective  allowable  limit.  If  the  process  average  exceeds  this 
limit,  we  say  that  the  contractor's  inspection  decision  process  is  out 
of  control,  and  the  contractor  will  be  directed  to  take  corrective  action 
to  bring  his  process  under  control. 

Returning  now  to  the  discussion  of  Facility  A,  it  is  emphasized  that 
decision  verification  procedures  have  not  been  used  at  Facility  A  to 
develop  a  process  average  for  use  in  controlling  the  contractor's  decision 
inspection  process;  rather  the  results  of  inspections  are  related  to 
product  quality.  Requests  to  the  contractor  for  corrective  action  have 
apparently  been  based  on  the  discovery  of  individual  material  defects 
during  inspection  rather  than  on  a  determination  that  the  contractor's 
inspection  decision  process  is  out  of  control.  If  this  is  the  case,  it 
supports  the  findings  of  the  Joint  Service  Panel  on  Contract  Administra- 
tion, whose  report  of  March  1971  said: 
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Further,  it  appears  to  the  Panel,  that  as  constituted,  the  CDV 
procedures  serve  only  to  assure  the  correction  of  specific  defi- 
ciencies detected  solely  on  a  random  basis.  Thus,  if  a  QAR  in 
performing  50  random  observations'  detects  a  particular  product 
or  process  deficiency,  he  may  direct  corrective  action  to  the 
particular  deficiency  which  he  encountered.  The  QAR  therefore  has 
little  way  of  knowing  that  other  characteristics  have  turned  de- 
defective  until  this  point  turns  up  in  a  later  inspection. 

All  of  this  seems  to  add  up  to  the  fact  that  Contractor  Decision  Verifi- 
cation Procedures  are  being  used  as  a  product  inspection  technique 
rather  than  as  a  method  of  controlling  the  contractor's  decision  process, 

C.   FINDINGS  AT  FACILITY  B 

The  visit  to  Facility  B  included  interviews  with  Government  QA  per- 
sonnel but  not  the  Naval  Plant  Representative. 

Personnel  with  whom  decision  verification  was  discussed  appeared  to 
be  enthusiastic  about  this  concept  as  a  quality  assurance  tool,  but  it 
was  soon  determined  that  Contractor  Decision  Verification  was  not  done 
continually  at  Facility  B.  It  is  done  only  during  periodic  quality- 
assurance  audits.  This  appears  primarily  to  result  from  a  lack  of 
sufficient  personnel  to  accomplish  decision  verification  continuously 
at  Hhe   nine  control  centers  which  have  been  established. 

Quality  assurance  audits  are  undertaken  on  an  as  needed  basis  such 
as  when  quality  data  indicates  an  adverse  quality  trend  developing  or 
on  a  180  day  cycle.  During  the  audit,  one  quality  assurance  representa- 
tive works  full  time  on  decision  verification. 

Like  Facility  A,  the  contractor  at  Facility  B  also  has  commercial 
production  in  progress  along  with  military  production  in  the  same 
general  location.  When  Decision  Verification  is  done  during  quality 
assurance  audits,  only  military  hardward  is  used  if  enough  of  it  is 
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available.  Quality  assurance  personnel  attempt  to  stay  away  from  any 
location  which  is  primarily  commercial  production.  If  an  item  of  com- 
mercial hardware  does  happen  to  come  through  a  predominantly  military 
product  station,  and  no  military  material  is  available  for  inspection, 
the  Quality  Assurance  Representative  is  permitted  to  use  the  commer- 
cial product  for  decision  verification  purposes. 

D.  FINDINGS  AT  FACILITY  C 

Facility  C  is  a  Naval  District  Supervisor  of  Shipbuilding,  Conver- 
sion and  Repair  office,  operating  under  the  direction  of  the  Naval  Sea 
Systems  Command.  The  field  visit  to  Facility  C  was  somewhat  limited 
in  scope,  but  interviews  with  QA  supervisory  personnel  were  conducted. 

Quality  assurance  management  personnel  at  Facility  C  are  familiar 
with  the  decision  verification  procedure  prescribed  by  the  Naval  Sea 
Systems  Command.  Recall  that  this  procedure  is  called  Verification  of 
Contractor  Records,  and  that  the  Naval  Sea  Systems  Command  makes  the 
inclusion  of  this  procedure  in  any  quality  assurance  program  optional. 
At  Facility  C  the  quality  assurance  personnel  have  elected  not  to  use 
decision  verification,  although  their  internal  directives  contain  com- 
prehensive instructions  dealing  with  the  accomplishment  of  Verification 
of  Contractor  Records. 

Construction  of  two  large  auxiliary  ships  for  the  Navy  in  a 
civilian  yard  is  one  of  the  major  production  activities  now  under  the 
cognizance  of  this  office.  This  construction  provides  an  opportunity 
for  application  of  decision  verification  procedures.  At  the  highest 
level  of  construction  activity,  16  quality  assurance  people  will  be 
assigned  directly  to  the  project  office  with  additional  people  available 
as  needed  from  the  Naval  District  SUPSHIP  office.  The  management  plan 
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for  these  ships  specifies  quality  assurance  tasks  to  be  performed. 
These  tasks  include  review  and  evaluation  of  contractor's  procedures 
and  processes,  and  product  verification  of  hardware,  but  decision 
verification  is  not  included. 

Facility  C  personnel  gave  two  reasons  for  not  using  Verification 
of  Contractor  Records  procedures .  First ,  the  type  of  operations  in- 
volved in  ship  construction  make  it  difficult  to  establish  control 
centers  and  verification  stations.  There  often  is  not  enough  hardware 
at  a  given  station  to  permit  the  required  number  of  observations. 
Second,  the  geographical  distribution  of  activities  in  a  shipyard 
makes  it  difficult  for  Government  representatives  to  be  present  at 
any  given  time  when  contractor- inspected  material  is  available  to 
be  used  for  decision  verification.  The  shortage  of  personnel  exists 
at  other  facilities  as  well.  There  are  just  not  enough  personnel  to 
adequately  carry  out  decision  verification  procedures. 

E.  NASA  AND  DECISION  VERIFICATION 

The  final  field  visit  was  to  a  contractor's  plant  in  which  a  NASA 
Space  Shuttle  is  being  built.  At  this  plant,  the  Defense  Contract 
Administration  Service  exercises  the  contract  administration  function 
under  a  letter  of  delegation  from  NASA.  According  to  NASA  directives, 
the  delegated  agency  is  to  use  its  own  administrative,  operational, 
and  procedural  instructions,  and  technical  documents  plus  any  addi- 
tional direction  which  NASA  provides.  If  the  agency  instructions  con- 
flict with  NASA  directives,  the  latter  prevail  [4] . 

For  inspection  purposes  there  are  two  important  categories  of 
characteristics  mentioned  in  NASA  contracts.  These  are  "mandatory" 
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and  "other  than  mandatory".  A  mandatory  characteristic  is  one,  which 

if  defective,  could  prevent  the  article  from  performing  its  intended 

purpose  in  the  next  higher  assembly  or  result  in  hazardous  or  unsafe 

conditions.  "Other  than  mandatory"  characteristics  are  those  selected 

for  inspection  and  test  "as  necessary  to  prevent  degradation  of 

quality"  [M-] .  Each  mandatory  characteristic  is  subject  to  inspection 

or  test  on  every  article  of  product.  "Other  than  mandatory" 

characteristics  may  be  inspected  by  sampling  when  appropriate. 

At  the  plant  visited,  the  contract  is  administered  by  the  Defense 

Contract  Administration  Service  which  has  long  since  discontinued  use 

of  decision  verification.  However,  NASA  contracts  could  also  be 

administered  by  Navy  offices  which  still  include  decision  verification 

procedures  as  an  element  in  quality  assurance.  NASA's  policy  on  this 

is: 

The  use  of  "Contractor  Decision  Verification  (CDV)"  or  similar 
systems  as  a  substitute  for  "mandatory"  or  "other  than  mandatory" 
inspection  of  an  article  is  prohibited. 

Contractor  Decision  Verification  (CDV)  or  any  similar  system 
is  not  to  be  substituted  for  this  requirement  nor  is  it  to  be 
used  for  product  inspection  or  article  acceptance . 

NASA  policy  as  stated  in  the  foregoing  quotations  has  been  inter- 
preted by  some  people  to  mean  that  CDV  or  similar  procedures  will  not 
be  used  in  administration  of  NASA  contracts. 

Obviously  the  four  activities  contacted  are  only  a  limited  sample 
of  the  total.  Nevertheless  they  are  representative  of  the  variety 


National  Aeronautics  and  Space  Administration,  Quality  Assurance 
Provisions  for  Government  Agencies,  NHB5300.1+(2B) (Washington:  Nov. 
1971),  p.  3-3. 
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of  production  activity  under  various  contract  administration  offices. 
This  chapter  has  shown  that  at  least  seme  activities  decision  veri- 
fication is  not  used.  The  following  chapter  will  discuss  these  veri- 
fication procedures  from  the  manager's  aspect. 
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V.  SOME  COMMENTS  FOR  CONTRACT  ADMINISTRATORS 
ON  USING  DECISION  VERIFICATION  PROCEDURES 

Previous  chapters  have  discussed  specified  procedures  for  determin- 
ing whether  or  not  a  contractor's  inspection  decision  process  is  oper- 
ating within  Government  established  limits.  This  chapter  discusses 
some  ways  a  Government  contract  administrator  might  locally  modify  the 
specified  procedures  in  order  to  make  them  more  useful  in  situations 
where  there  is  too  little  production  volume  to  feasibly  obtain  the 
specified  number  of  sample  observations.  Operating  characteristic 
curves  are  presented  to  show  how  variables  such  as  sample  size,  fre- 
quency of  sampling,  and  actual  process  fraction  defective  affect  the 
usefulness  of  decision  verification.  Also  discussed  are  some  human 
factors  which  impact  on  the  operation  of  decision  verification. 

A.   SAMPLING  AND  DEVELOPMENT  OF  CONTROL 
CHARTS  FOR  VERIFICATION 

Since  100  percent  inspection  of  each  item  is  often  not  practical  or 
not  considered  cost  effective,  Navy  quality  assurance  personnel  use 
sampling  techniques  in  performing  decision  verification.  When  sampling 
is  done,  there  is  always  a  risk  that  unsatisfactory  inspection  decision 
performance  will  be  accepted  by  the  Government.  This  risk  arises  from 
the  fact  that  the  sample  may  not  be  representative  of  the  process  quality 
and  from  errors  in  inspection  of  decisions  from  the  sample. 

In  practice,  not  even  100%  inspection  always  insures  a  defect  free 
product  or  process.  Experiments  with  100%  inspection  indicate  that  the 
average  number  of  defects  detected  may  be  considerably  less  than  100% 
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of  the  defects  present.  An  experiment  which  was  done  to  determine  the 
influence  of  complexity  on  inspection  errors  resulted  in  defect  detec- 
tions of  from  30%  for  complex  items  to  80%  for  less  complex  items  [10] . 
This  of  course  refers  to  individual  inspectors  rather  than  to  the  con- 
tractor's inspection  process  as  a  whole.  The  contractor's  process 
would  be  expected  to  have  a  higher  effectiveness  because  of  repetitive 
inspections  and  other  controls  on  quality. 

Since  100%  inspection  may  not  guarantee  that  all  defective  decisions 
will  be  discovered,  the  manager  should  realize  that  sampling  is  even 
less  likely  to  uncover  all  errors.  It  follows  that  since  decision  veri- 
fication uses  sampling j  we  should  expect  that  not  all  incorrect  decisions 
of  a  contractor  will  be  discovered. 

Although  the  two  verification  methods  discussed  in  this  paper  use 
sampling,  there  are  important  differences  between  them.  One  method  is 

based  on  using  a  control  chart  approach  to  determine  when  the  contractor's 

...  9 

decision  process  is  out  of  control.   The  other  method  appears  simply  to 

sample  the  inspection  decisions,  and  then  accept  or  reject  the  contrac- 
tor's inspection  process  by  comparing  the  percent  defective  in  the 
sample  with  2.8%. 

Contractor  Decision  Verification  procedures  are  based  on  the  concept 
of  a  control  chart  called  a  "p-chart"  which  is  designed  to  determine 
whether  or  not  the  process  is  under  control.  The  p-chart  shows  variation 


9  ... 

The  control  chart  is  a  useful  device  for  determining  whether  ob- 
served changes  are  due  simply  to  chance  fluctuations  or  to  actual 
changes  in  the  process  because  of  sucn  things  as  deterioration  of 
machine  parts,  or  in  this  case  to  mistakes  by  the  inspector.  A  com- 
prehensive discussion  of  control  charts  can  be  found  in  Quality  Control 
and  Industrial  Statistics  by  A.  J.  Duncan  [Ref .  13] . 
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around  a  central  value  of  the  fraction  defective  in  the  output  of  a 
process.  In  this  case  it  shows  variations  in  the  fraction  defective 
of  inspection  decisions. 

Under  the  assumption  that  most  contractor's  inspection  decision 
processes  are  accurate  99%  of  the  time,  the  Government  has  established 
an  Acceptable  Quality  Level  (AQL)  of  1%  for  contractor  inspection 
decisions  [11] .  This  AQL  value  is  used  as  the  central  line  of  the 
p-charts  for  each  Product  Control  Center  when  Contractor  Decision  Veri- 
fication is  applied  as  an  element  of  a  quality  assurance  program  [7] . 
The  Upper  Control  Limit  (UCL)  for  the  chart  is  set  at  three  standard 
deviations,  above  the  AQL,  or  the  so-called  "three-sigma  limit".  The 
use  of  a  three-sigma  limit  significantly  reduces  the  probability  that 
any  given  sample  result  which  falls  outside  this  limit  does  so  by 
chance  alone  if  the  process  is  under  control. 

In  order  to  illustrate  how  the  p-chart  could  be  prepared  using  Navy 
procedures  for  Contractor  Decision  Verification,  we  consider  the 
initial  step  in  verification  of  taking  50  observations  for  five  days 
to  get  a  total  of  250  observations.  The  central  line  fraction  defective 
p'  is  the  AQL  value  of  0.01.  In  the  example,  the  three-sigma  limit  can 
be  computed  as  follows: 


slsna  =  ^SHT     =  J. 


(0.01)(0.99T~     _  n  nc 
251 "  °'06 


Three-sigma  =  0.018 

The  Upper  Control  Limit  for  this  p-chart  is  equal  to  the  central 
line  value  of  0.01  plus  the  value  of  three-sigma,  and  thus  the  Upper 
Control  Limit  in  this  case  is  0.028.  The  acceptance  number  for  the 


41 


sample  of  250  is  obtained  by  multiplying  the  sample  size  by  the  value 
of  the  Upper  Control  Limit.  The  acceptance  number  for  the  example  is 
7.  Thus  if  7  or  fewer  defective  decisions  are  found  in  a  sample  of 
250,  the  contractor's  inspection  decision  making  process  is  considered 
to  be  under  control. 

Continuing  with  this  example,  if  the  process  is  under  control  after 
the  initial  sample,  only  50  observations  are  required  each  11  day 
interval  thereafter  as  long  as  the  process  remains  under  control  [7]  . 
The  p-chart  for  the  50  observations  is  similar  to  the  previous  chart. 
The  central  line  is  at  0.01,  but  the  control  limit  is  different.  Since 
the  standard  deviation  varies  inversely  with  the  square  root  of  the 
sample  size,  the  Upper  Control  Limit  gets  further  away  from  the  central 
line  value  as  the  sample  size  is  decreased.  In  this  case  the  Upper 
Control  Limit  is  now  0.052.  Based  on  this  upper  limit,  the  process  is 
considered  under  control  unless  there  are  more  than  three  defectives 
in  a  sample  of  50  observations. 

Contractor  Decision  Verification  procedures  and  Verification  of 
Contractor  Records  have  the  same  objective,  but  there  are  significant 
differences  in  these  procedures.  We  have  seen  that  Contractor  Decision 
Verification  procedures  make  use  of  the  concepts  of  a  p-chart  and  that 
the  maximum  allowable  fraction  defective  of  the  sample  varies  with  the 
sample  size.  This  is  not  true  with  procedures  for  Verification  of 
Contractor's  Records,  which  do  not  make  use  of  control  charts  nor  vary 
the  maximum  allowable  fraction  defective  with  sample  size. 

In  accomplishing  Verification  of  Contractor  Records,  a  sample  of 
unspecified  size  is  taken  from  the  contractor's  inspection  decisions. 
Regardless  of  the  sample  size,  the  contractor's  process  is  considered 
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unacceptable  if  the  fraction  defective  of  the  sample  exceeds  2.8%. 
Using  these  procedures,  the  individual  contract  administrator  has 
considerable  flexibility  in  selecting  a  sample  size  commensurate  with 
the  volume  of  inspection  decisions  available  instead  of  being  re- 
stricted to  samples  of  50,  100,  or  250  as  in  Contractor  Decision  Veri- 
fication procedures. 

This  section  has  briefly  discussed  how  sample  results  may  be  used 
in  making  the  decision  to  accept  or  reject  the  contractor's  process, 
but  it  has  not  given  the  manager  any  guidance  for  application  of 
decision  verification  procedures.  The  next  section  will  discuss  one 
tool  the  contract  adndnistration  manager  can  use  in  deciding  whether 
or  not  a  modified  decision  verification  procedure  could  be  productively 
included  in  his  quality  assurance  program,  even  if  any  requirement  to 
accomplish  decision  verification  had  been  waived  because  of  lack  of 
adequate  production  volume. 

B.   OPERATING  CHARACTERISTIC  CURVES 
FOR  DECISION  VERIFICATION 

In  determining  the  applicability  and  feasibility  of  decision  veri- 
fication in  his  particular  situation,  a  manager  should  consider  a 
number  of  factors.  These  include  practical  sample  size,  frequency  of 
sampling,  cost  of  inspection  versus  cost  of  material  rejected  further 
along  in  the  process,  and  other  inspections,  such  as  product  verifica- 
tion inspection,  which  may  reduce  the  need  for  decision  verification. 

An  operating  characteristic  curve  is  one  device  available  to  the 
manager  as  an  aid  in  determining  the  potential  value  of  decision 
verification  procedures  in  his  overall  quality  assurance  program.  The 
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operating  characteristic  curve  shows  how  the  probability  of  acceptance 
varies  with  the  quality  of  decision. 

Figure  1  shows  operating  characteristic  curves  for  p- charts  which 
could  be  used  in  Contractor  Decision  Verification  Procedures.  Three 
of  these  curves,  namely  50,  100,  and  250  are  those  which  should  be  used 
when  doing  verification  as  specified  by  the  directive.  Each  curve  is 
plotted  for  a  different  sample  size.  The  probability  of  the  results  of 
the  verification  procedure  being  considered  acceptable  for  a  sample  of 
size  (n)  is  plotted  against  the  process  fraction  defective. 

Turning  to  Figure  1  it  can  be  seen  that  for  any  particular  value  of 
process  fraction  defective,  with  an  AQL  of  0.01,  a  sample  size  of  25 
gives  less  probability  of  detecting  an  out  of  control  process  than  a 
sample  size  of  50.  The  manager  can  use  the  operating  characteristic 
curves  to  determine  how  much  better  chance  of  detecting  a  shift  in  the 
process  exists  when  the  sample  is  50  instead  of  25.  Doubling  the  sample 
size  generally  does  not  double  the  probability  of  detecting  a  process 
shift.  The  manager  should  compare  the  value  of  an  added  increment  of 
protection  against  undetected  process  shifts  to  the  cost  of  getting  that 
additional  protection. 

Continuing  with  Figure  1,  if  the  actual  fraction  defective  of  the 
process  shifts  to  5%,  there  is  about  a  0.14  probability  that  this  shift 
in  the  process  will  be  detected  if  a  sample  of  25  is  taken.  Doubling 


For  a  detailed  discussion  of  operating  characteristic  curves,  see 
Quality  Control  and  Industrial  Statistics  by  A.  J.  Duncan.  Chapter  7 
and  Chapter  19  of  this  text  are  of  particular  value  in  understanding 
operating  characteristic  curves  applicable  to  decision  verification 
procedures . 
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Figure  1.  Operating  characteristic  curves  for  single  limit  p-charts 
for  Contractor  Decision  Verification. 
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the  sample  size  to  50,  gives  a  0.24  probability  that  the  process  shift 
will  be  detected.  If  these  same  sample  sizes  are  used  to  verify  the 
process  at  11  day  intervals  as  specified  in  procedures  for  Contractor 
Decision  Verification,  it  is  possible  to  calculate  that  there  is  an 
approximate  0.60  probability  that  the  shift  will  still  be  undetected 
after  about  30  days  if  the  sample  size  is  25.  This  probability  drops 
to  approximately  0.40  if  the  sample  size  is  50.  Since  the  samples  are 
random,  the  probability  of  not  detecting  a  shift  after  a  given  number 
of  samples  is  the  product  of  the  individual  sample  probabilities  of 
not  detecting  the  shift. 

Figure  2  shows  operating  characteristics  curves  for  sample  inspec- 
tion plans  which  demonstrate  the  performance  of  the  Verification  of 
Contractor  Records  procedures  where  the  sample  size  is  arbitrary. 
Curves  for  various  sample  sizes  are  shown.  It  can  readily  be  seen 
again  that  the  sample  size  affects  the  probability  of  accepting  the 
contractor's  inspection  process  for  any  particular  process  fraction 
defective . 

The  manager  can  use  these  curves  in  much  the  same  way  as  he  could 
use  those  in  Figure  1.  From  the  various  curves  he  can  determine  how 
the  probability  of  accepting  the  contractor's  process  varies  with 
sample  size. 

It  is  interesting  to  compare  some  operating  characteristic  curves 
for  Contractor  Decision  Verification  and  those  developed  here  for  use 
in  Verification  of  Contractor  Records.  Figure  3  shows  some  operating 
characteristic  curves  for  Contractor  Decision  Verification  and  for 
Verification  of  Contractor  Records  for  the  same  sample  sizes.  It  can 
easily  be  seen  that  when  both  approaches  use  the  same  sampling  size, 
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Figure  2.  Operating  characteristic  curves  for  single  sample  inspection 
plans  for  Verification  of  Contractor  Records. 
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Figure  3.  Comparison  of  some  operating  characteristic  curves  for 
Contractor  Decision  Verification  and  for  Verification 
of  Contractor  Records. 
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Verification  of  Contractor  Records  procedures  give  a  greater  probability 
of  rejecting  the  contractor's  process  for  unacceptable  process  fraction 
defectives.  This  is  true  up  to  a  sample  size  of  250  at  which  point  the 
operating  characteristic  curves  for  Contractor  Decision  Verification 
and  for  Verification  of  Contractor  Records  are  identical.  This  means 
that  for  samples  of  less  than  250  observations,  the  decision  verification 
procedures  used  by  activities  under  the  Naval  Sea  Systems  Command  perform 
the  decision  verification  function  with  greater  statistical  effectiveness 
than  procedures  specified  by  Naval  Air  Systems  Command. 

In  spite  of  the  fact  that  either  of  the  decision  verification  pro- 
cedures discussed  in  this  paper  can  be  useful  in  a  quality  assurance 
program  neither  of  them  was  found  to  be  extensively  used.  The  next 
section  discusses  some  possible  reasons  for  this  non-use. 

C.  PRACTICAL  PROBLEMS  OBSTRUCTING  THE 
USE  OF  DECISION  VERIFICATION 

Decision  verification  procedures  were  developed  to  aid  the  Government 
in  measuring  the  effectiveness  of  a  contractor's  quality  assurance  pro- 
gram. In  practice  these  procedures  have  evidently  not  produced  the 
desired  results.  They  have  been  criticized  and  even  deleted  from  some 
quality  assurance  manuals. 

People  are  the  focus  of  many  problems  associated  with  decision  veri- 
fication. In  many  cases  there  are  not  enough  quality  assurance  personnel 
to  do  all  of  the  assigned  tasks,  including  decision  verification.  If 
enough  confidence  can  be  generated  in  decision  verification,  a  manage- 
ment decision  might  result  in  more  time  spent  on  this  type  of  inspection. 
Quite  the  contrary  appears  to  be  happening  today. 

The  value  and  the  feasibility  of  decision  verification  procedures 
is  being  seriously  questioned  by  the  users  of  these  procedures  and  by 
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various  study  panels  on  procurement.  There  is  some  merit  to  the  argu- 
ments against  decision  verification.  For  example,  looking  at  the 
operating  characteristic  curve  in  Figure  1  for  a  sample  size  of  250, 
one  sees  that  a  process  fraction  defective  three  times  as  large  as  the 
Acceptable  Quality  Level  has  approximately  a  50%  chance  of  being 
accepted.  It  is  little  wonder  that  some  people  have  lost  confidence 
in  decision  verification.  On  the  other  hand  decision  verification  in 
some  cases  may  offer  all  the  assurance  of  inspection  quality  necessary. 
For  example,  using  the  same  sample  size  of  250,  if  a  manager  is  willing 
to  accept  some  probability  that  the  fraction  defective  can  be  as  much 
as  but  not  exceed  a  fraction  defective  of  0.06,  he  may  confidently  use 
decision  verification  procedures,  since  the  operating  characteristic 
curve  shows  that  for  a  sample  of  250,  the  probability  is  almost  zero 
of  accepting  decision  processes  which  have  fractions  defective  of  0.06 
or  greater. 

There  is  some  concern  about  the  motivation  of  quality  assurance 
personnel  to  do  accurate  decision  verification.  Several  experienced 
Government  supervisory  personnel  with  whom  this  was  discussed  felt 
that  the  quality  assurance  representative  was  often  reluctant  to  per- 
form decision  verification  because  he  had  to  judge  the  performance  of 
personnel  rather  than  performance  of  a  manufacturing  process.  This 
problem  could  be  especially  acute  in  plants  where  Government  quality 
assurance  personnel  have  had  long  association  with  the  contractor's 
inspectors . 

If  a  quality  assurance  representative  is  less  than  conscientious 
he  can  easily  falsify  his  records  of  decision  verification  performance. 
The  DD  1711  form  used  to  record  verification  lists  the  stations  veri- 
fied, but  specific  hardware  items  inspected  are  not  identified  unless 
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a  defective  observation  is  made.  The  audit  trail  for  decision  verifi- 
cation is  inadequate.  Thus  we  see  that  attempts  to  accomplish  decision 
verification  may  well  have  to  overcome  several  obstacles,  not  the  least 
of  which  is  in  some  cases  the  possible  lack  of  motivated  and  energetic 
quality  assurance  representatives. 

This  chapter  has  discussed  the  idea  of  control  charts  used  in  veri- 
fication, and  has  noted  differences  in  two  Navy  decision  verification 
procedures.  The  value  of  operating  characteristic  curves  to  the 
manager  has  been  illustrated.  The  final  chapter  of  this  paper  will 
present  conclusions  and  recommendations  arising  from  this  research. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 

Several  conclusions  have  been  reached  as  a  result  of  this  investi- 
gation into  decision  verification  procedures  used  by  various  contract 
administration  offices.  These  can  be  conveniently  grouped  for  dis- 
cussion into  general  and  specific  categories. 

A.  GENERAL  CONCLUSIONS 

Some  method  of  assessing  the  effectiveness  of  a  contractor's  inspec- 
tion decisions  would  be  useful  in  the  overall  Government  quality 
assurance  program,  but  it  is  not  an  essential  element.  The  objectives 
of  the  Government's  quality  program  can  be,  and  most  often  are,  met 
without  recourse  to  decision  verification.  There  is  little  emphasis  on 
decision  verification  in  the  performance  of  many  Navy  contracts. 

Although  statistically  sound,  decision  verification  has  been  sub- 
jected to  some  analysis  in  the  past  [Ref.  14].  It  is,  however,  the 
practical  problems  associated  with  accomplishing  decision  verification 
rather  than  anything  else  which  have  caused  decision  verification  to 
be  neglected. 

B.  SPECIFIC  CONCLUSIONS 

Quality  assurance  directives  issued  by  Naval  Air  Systems  Command 
and  Naval  Sea  Systems  Command  for  use  by  field  contract  administration 
activities  do  not  explain  decision  verification  in  sufficient  statisti- 
cal and  probabilistic  detail.  They  do  not  give  the  manager  enough 
information  about  decision  verification  procedures  to  enable  him  to 
make  a  sound  decision  on  whether  to  use  these  procedures  as  prescribed; 
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to  modify  them  to  accommodate  local  conditions;  or  not  to  use  decision 
verification  at  all. 

Some  activities  do  not  use  decision  verification  procedures  con- 
tinuously because  there  is  not  enough  production  volume  to  support  con- 
tinuous verification.  Instead,  they  only  include  decision  verification 
in  the  quality  assurance  audit  program.  Product  Control  Centers  may 
be  audited  for  10  day  periods  at  approximately  six-month  intervals. 

Audits  can  produce  little  useful  information  as  far  as  decision 
verification  is  concerned.  As  an  example  of  this,  if  40  observations 
a  day  are  made  for  two  weeks,  during  the  audit  there  is  about  a  30% 
probability  that  the  contractor's  process  would  be  accepted  as  in 
control  even  if  the  process  fraction  defective  were  3%  instead  of  the 
Acceptable  Quality  Level  value  of  1%.  If  low  production  volume  was 
the  initial  cause  for  doing  decision  verification  only  on  an  audit 
basis,  it  is  doubtful  that  even  as  many  as  40  observations  per  day 
would  be  available.  Smaller  samples  than  40  observations  would  give 
an  even  greater  probability  that  the  process  was  in  control  when  it 
actually  was  not. 

People  are  definitely  a  critical  element  in  the  practical  appli- 
cation of  decision  verification.  Both  quality  and  quantity  impact 
on  whether  or  not  decision  verification  will  be  successful.  The 
quality  assurance  representatives  must  actually  perform  the  verifica- 
tion, and  the  supervisors  and  managers  must  have  confidence  in  the 
results.  This  investigation  suggests  that  in  some  cases  neither  event 
occurs,  because  results  of  decision  verification  as  currently  recorded 
do  not  lend  themselves  to  easy  audit. 
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C.  RECOMMENDATIONS 

Because  of  problems  encountered  in  performing  decision  verification 
and  lack  of  confidence  in  the  results,  there  seems  to  be  mixed  interest 
among  quality  assurance  personnel  in  continuing  to  use  decision  verifi- 
cation in  a  quality  assurance  program.  Perhaps  the  usefulness  and  re- 
quirement for  decision  verification  should  be  reviewed  by  the  material 
commands  of  the  Navy  and  a  determination  made  as  to  whether  or  not  this 
decision  verification  should  be  retained  as  an  element  of  Navy  quality 
assurance. 

Further  research  is  needed  to  determine  the  optimum  mix  of  the 
various  elements  of  quality  assurance  such  as  product  verification, 
procedures  evaluation,  and  decision  verification.  The  possibility  of 
developing  a  more  flexible  procedure  for  use  by  the  manager  in  applying 
decision  verification  should  be  investigated.  This  method  might  include 
such  variables  as  volume  of  production,  number  of  personnel  available, 
value  of  the  product,  and  best  location  of  various  Product  Control 
Centers  in  the  production  flow. 

There  are  some  recommendations  of  a  more  immediate  nature.  For  in- 
stance, activities  which  now  accomplish  decision  verification  only  on 
an  audit  basis  might  consider  discontinuing  this  practice.  Because 
of  the  infrequent  sampling,  the  information  obtained  about  the  status 
of  the  contractor's  inspection  decision  process  does  not  appear  to  be 
worth  the  full  time  effort  of  a  quality  assurance  representative  during 
the  period  of  the  audit. 

It  is  also  suggested  that  decision  verification  procedures  be 
standardized  throughout  the  Navy  if  they  are  retained  as  part  of  the 
Navy  quality  assurance  program. 
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.  Finally  it  is  proposed  that  managers  at  the  local  plant  level 
have  their  people  develop  operating  characteristic  curves  such  as 
those  in  this  paper.  These  curves  can  then  be  used  in  determining 
the  value  of  doing  verification  using  various  sample  sizes  and 
sampling  frequencies  which  are  feasible  at  the  individual  production 
activity.  The  local  manager  should  decide  whether  or  not  he  can 
effectively  use  decision  verification  procedures. 

The  principal  usefulness  of  this  work  is  in  bringing  into  focus 
the  issues  to  consider  and  assess  concerning  the  viability  of  decision 
verification  processes  at  a  given  plant. 

In  this  thesis  we  have  discussed  some  methods  of  accomplishing 
decision  verification  as  related  to  quality  assurance.  We  have  also 
looked  at  some  of  the  problems  which  contract  administrators  face  when 
trying  to  accomplish  decision  verification  under  conditions  which  are 
not  favorable  to  using  this  quality  assurance  technique. 

It  is  hoped  that  this  thesis  effort  will  be  useful  to  those  con- 
tract administrators  who  wish  to  include  decision  verification  as  an 
element  in  their  quality  assurance  program. 
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